Three dimensional volumetric display input and output configurations

ABSTRACT

The present invention is a system that allows a number of 3D volumetric display or output configurations, such as dome, cubical and cylindrical volumetric displays, to interact with a number of different input configurations, such as a three-dimensional position sensing system having a volume sensing field, a planar position sensing system having a digitizing tablet, and a non-planar position sensing system having a sensing grid formed on a dome. The user interacts via the input configurations, such as by moving a digitizing stylus on the sensing grid formed on the dome enclosure surface. This interaction affects the content of the volumetric display by mapping positions and corresponding vectors of the stylus to a moving cursor within the 3D display space of the volumetric display that is offset from a tip of the stylus along the vector.

CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application is related to and claims priority to U.S.provisional application entitled User Interfaces For VolumetricDisplays, having serial No. 60/350,952 (S&H Docket 1252.1054P), byKurtenbach et al, filed Jan. 25, 2002, this application is also relatedto U.S. application entitled Volume Management System For VolumetricDisplays, having Ser. No. ______ (S&H Docket 1252.1065), by Kurtenbachet al, filed herewith, to U.S. application entitled Widgets DisplayedAnd Operable On A Surface Of A Volumetric Display Enclosure, having Ser.No. ______ (S&H Docket 1252.1066) by Fitzmaurice et al, filedconcurrently herewith, to U.S. application entitled Graphical UserInterface Widgets Viewable And Readable From Multiple Viewpoints In AVolumetric Display, having Ser. No. ______ (S&H Docket 1252.1067), byFitzmaurice et al, filed concurrently herewith, to U.S. applicationentitled A System For Physical Rotation of Volumetric Display EnclosuresTo Facilitate Viewing, having Ser. No. ______ (S&H Docket 1252.1068), byBalakrishnan et al, filed concurrently herewith, to U.S. applicationentitled Techniques For Pointing To Locations Within A VolumetricDisplay, having Ser. No. ______ (S&H Docket 1252.1069), by Balakrishnanet al, filed concurrently herewith, and all of which are incorporated byreference herein.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention is directed to input and outputconfigurations for three-dimensional volumetric displays and, moreparticularly, to input configurations that allow the content of athree-dimensional volumetric display output configuration to be affectedby actions by a user operating within an input configuration.

[0004] 2. Description of the Related Art

[0005] A class of three-dimensional (3D) displays, called volumetricdisplays, is currently undergoing rapid advancement. The types ofdisplays in this class include holographic displays, swept volumedisplays and static volume displays. Volumetric displays allow forthree-dimensional (3D) graphical scenes to be displayed within a true 3Dvolume. Such displays can take many shapes including cylinders, globes,domes, cubes, an arbitrary shape, etc. with a dome being a typicalshape. Because the technology of these displays is undergoing rapiddevelopment those of skill in the art are concentrating on theengineering of the display itself. As a result, the man-machineinterface to or input/output configurations with which people interfacewith these types of displays is receiving scant attention.

[0006] While the volumetric displays allow a user to view differentparts of a true 3D scene, the act of viewing the different partstypically requires that the user physically move around (or over) thedisplay or that the display be moved or rotated in front of the user. Asthe display moves relative to the user, graphical objects may also moverelative to the user. When the display is relatively stationary or whenit is relatively moving, the user may need to interact with the display.As a result, what the user needs is an effective mechanism forinteracting with the display.

SUMMARY OF THE INVENTION

[0007] It is an aspect of the present invention to provide effectivemechanisms for a user to interact with content of the three-dimensionalvolumetric display.

[0008] It is also an aspect of the present invention to provide inputand output configurations for a three-dimensional volumetric display.

[0009] It is another aspect of the present invention to provide dome,cubical and cylindrical output configurations.

[0010] It is also an aspect of the present invention to provide inputconfigurations that allow a 3D volumetric input space to be mapped tothe 3D volumetric display, a planer 2D input space to be mapped to the3D volumetric display, a planar 2D input space to be mapped to a planar2D space within the 3D volumetric display, and a non-planar 2D inputspace to be mapped to the 3D volumetric display.

[0011] The above aspects can be attained by a system that allows anumber of 3D volumetric display configurations, such as dome, cubicaland cylindrical volumetric display enclosures, to interact with a numberof different input configurations, for example, a three-dimensionalposition sensing system, a planar position sensing system and anon-planar position sensing system. The user interacts with the inputconfigurations, such as by moving a stylus on a sensing grid formed onan enclosure surface. This interaction affects the content of thevolumetric display, for example, by moving a cursor within the 3Ddisplay space of the volumetric display.

[0012] These together with other aspects and advantages which will besubsequently apparent, reside in the details of construction andoperation as more fully hereinafter described and claimed, referencebeing had to the accompanying drawings forming a part hereof, whereinlike numerals refer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013]FIG. 1 shows a volumetric display.

[0014]FIGS. 2, 3 and 4 depict a 3D to 3D system configurations.

[0015]FIGS. 5, 6 and 7 depict 2D to 3D configurations.

[0016]FIG. 8 shows a non-planar to 3D configuration.

[0017]FIGS. 9, 10, 11, 12, 13 and 14 show configurations with physicalintermediaries.

[0018]FIG. 15 depicts components of the system

[0019]FIGS. 16A, 16B, 16C and 16D illustrate digitizer embodiments.

[0020]FIGS. 17, 18A and 18B show a dome shaped digitizer.

[0021]FIG. 19 depicts the operations of the system.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Volumetric displays allow a user to have a true three-dimensional(3D) view of a scene 12 and are typically provided in the form of a dome14, as depicted in FIG. 1. The user 16, as can be surmised from FIG. 1,can move about the dome 14 to view different parts of the scene 12. Froma particular arbitrary viewpoint or position, a user may want tointeract with the scene or content within the volumetric display.

[0023] There are a number of different solutions to this problem. Thesesolutions involve creating input/output configurations for thevolumetric display that define a spatial correspondence between an inputspace and an output space. The configurations also define a dynamicallyupdatable spatial correspondence of the input and output spaces with theuser.

[0024] In a first solution, a 3D volumetric input space is mapped to a3D volumetric display space. In one configuration, as depicted in FIG.2, the user's hand 30 is tracked via a glove or a set of cameras in avolume 32 directly below the display volume 34. A virtual representationof the hand 36, or some other type of position indicator, such as acursor, is superimposed into the 3D output volumetric display 34. In asecond configuration, as depicted in FIG. 3, the 3D display 50 issurrounded by a 3D input space 52, created by a 3D volume input system,such as the Flock of Birds system from Ascension Technology Corporation.In this configuration, the user's hand 54, including a positionindicator/sensor, is mapped to a cursor 56 or some other positionindicator representation, such as a virtual hand, within the display 50.The position sensor also produces a vector that indicates whichdirection the sensor is pointing. The vector can be used to create acursor in the enclosure at a fixed position along the vector. Ratherthan using the vector produced by the position sensor, the system infersan input vector based on the position of the input device and the centerof the display. This spatial relationship or correspondence between theinput space, output space and user position is dynamically updated asthe user moves about the display. That is, the input/output space isautomatically compensated/reconfigured. Another configuration is to usehalf-silvered mirrors 70 (see FIG. 4) to combine the volumetric image 72with the user's view of their hands in a hand movement volume. This way,the user sees their hands operating within the display. Anotheralternative is to use a camera to capture the users hands in the inputspace and superimpose them onto the volumetric display space. Anotheralternative is an augmented-reality system where the user has asee-through, head mounted display (2D) which is being tracked. As theuser moves the position and orientation of their head, graphics arepresented on the LCD display and are aligned with real-world objects.

[0025] Another solution is to map a planer 2D input space into a 3Doutput space. This is particularly useful in controlling some subset ofthe 3D volumetric display space. For example, a standard 2D digitizingtablet or digitizer 90 (see FIG. 5) or a regular mouse can be mapped tocontrol aspects of the 3D scene, such as moving 3D objects along twodimensions.

[0026] A further solution is to map a planar 2D input space to a planar2D space within the 3D output space of the display, as depicted in FIG.6. In this situation, the system maps the input space of a digitizingtablet 110 and the tilt/orientation of the tablet as sensed by atilt/orientation sensor 112 to a corresponding planar space 114 in thedisplay 116. The angle of the plane 114 is responsive to the sensor 112.If the display enclosure 130 has planar surfaces (e.g., a cubicenclosure), the enclosure surface is used as the planar input device, asdepicted in FIG. 7. It is also possible to use a transparent digitizersuperimposed over an LCD display.

[0027] Still another solution is to map a non-planar 2D input space to a3D output space. In this solution, as depicted in FIG. 8, the systemuses the display enclosure 140 as the input space (i.e., the enclosureis a transparent digitizing input surface). In this embodiment, theposition 142 touched by the user or indicated by a pointing device, suchas a stylus or surface fitting curved mouse, is mapped to a position inthe display. This is a direct and compelling way to interact with thesedisplays.

[0028] In addition to direct manipulation using the hands, the solutionsdescribed herein also provide physical intermediaries between the handsand the input space as described below.

[0029] Another solution when the user desires to interact directly withthe enclosure surface, is to deform the surface 160 of a conventionaldeformable membrane surface that detects multiple pressure points andfinger pinches 162, as depicted in FIG. 9. In this situation, thesurface to display mapping discussed above is performed.

[0030] Instead of using the hands directly on the enclosure surface, thesystem has a surface that detects and tracks a variety of input devices.A digital stylus 180, as shown in FIG. 10, where a point and anorientation can be input or a Rockin'Mouse shaped device 190, as shownin FIG. 11 (see U.S. Pat. No. 6,115,028) also allowing a point and anorientation to be input are used. A surface fitting wireless mouse, suchas a curved (concave) bottom mouse, can be used with a curved surfaceoutput configuration. This type of mouse can also be park-able usingelectrostatic, magnetic or some other sticky method of removablyadhering the mouse to the display surface. Using a mouse has theadvantage of buttons and form factors with which people are familiar. Inthis situation, the surface to display mapping discussed above isperformed.

[0031] The physical intermediaries also do not have to be on theenclosure itself as described below.

[0032] In an embodiment input devices 200, such as buttons, keyboards,sliders, touch-pads, mice and space-ball type devices, etc., are mountedalong the perimeter of the display (see FIG. 12). In this embodiment,the input devices such as buttons for up, down, forward, backward, leftand right motions, allowing multiple degrees of freedom, are used tocontrol the position of a cursor like such buttons control the positionof a cursor in a 2D system. The input devices 210, 212, 214 may need tobe “repeated” (i.e., have more than one of each along the perimeter) toallow for simultaneous used by many users, or for use from any positionthe user may be standing/sitting at as shown in FIG. 13. Rather thathaving multiple input devices positioned around the display as depictedin FIG. 13, the mounting platform 220 that houses these devices could bemade moveable (rotatable) around the display, as depicted in FIG. 14, sothat users can easily bring the required device within reach by simplymoving the platform. These devices typically communicate wirelessly byradio or infrared signals. The position of the movable device alsoprovides information about the users position or viewpoint.

[0033] The present invention is typically embodied in a system asdepicted in FIG. 15 where physical interface elements 230, such as arotary dome position encoder, infrared user position detectors, akeyboard, touch sensitive dome enclosure surface, mouse, beam pointer,beam pointer with thumbwheel, stylus and digitizer pad or stylus andstylus sensitive dome enclosure surface, stylus with pressure sensor,flock-of-birds, etc. are coupled to a computer 232, such as a serverclass machine. The computer 232 uses a graphical creation process, suchas the animation package MAYA available from Alias|Wavefront, Inc., tocreate three-dimensional (3D) scene elements. This process, usingposition inputs from the input configurations as discussed herein, alsocreates the virtual interface elements, such as a virtual hand, a 3Dpoint cursor, a 3D volume cursor, a pointing beam, a bead, etc. Thedisplay output, including the scene and interface elements, is providedto a volumetric display apparatus configuration 234, such as one thatwill produce a 3D holographic display and discussed herein.

[0034] The configurations that include a transparent digitizer or touchsensitive surface have a number of different shapes as depicted in FIGS.16A-16D. In one embodiment a dome shaped enclosure 250 has a dome shapeddigitizing tablet as depicted in FIG. 16A. In another embodiment thedome shaped enclosure 256 (see FIG. 16B) is used with a rectangular orcylindrical shaped digitizing tablet 258. In a further embodiment, asshown in FIG. 16C, a cylindrical or cubical enclosure 260 is used withcylindrical or cubical digitizer surface. In a different embodiment theenclosure 264 is dome shaped (or cubical or cylindrical) and thedigitizing surface 266 is planar as depicted in FIG. 16D.

[0035] A digitizer 280 (see FIG. 17), such as described in U.S. Pat. No.5,854,449 incorporated by reference herein, determines a position of astylus or pointer 282 relative to a surface 284, such as a transparentdome surface, having a checker board type closely spaced positional grid286 thereon when seen from above. A processor 288 determines the coarseposition of the pointer relative to the grid by sampling the grid linesthrough a set of multiplexers 290 and 292. An error correction system294 generates and outputs a true position of the pointer 282 relative tothe surface 284 to a computer system 232 (see FIG. 15). The pointer 282typically includes an electromagnetic transducer for inducing a signalin the positional grid 286 and the processor 288 is coupled to thepositional grid 286 for sensing the signal and generating the coarseposition of the pointer 282. The transducers also allow thedetermination of a vector from grid signals that indicates in whichdirection the pointer 282 is pointing. Touch sensitive input surfacesoperate in a similar fashion.

[0036] The positional grid 286 can be applied to a surface of anenclosure, such as a dome shaped enclosure 310, as depicted in FIGS. 18Aand 18B. FIGS. 18A and 18B (an exploded view) show a section 312 of thedome surface including an inner substrate 314 and outer substrate 316between which is sandwiched the grid 318. The substrates comprisetransparent materials, such as glass or plastic.

[0037] In using these input and output configurations the computersystem 232 (see FIG. 15) performs a number of operations as depicted inFIG. 19. The operations include obtaining 330 the coordinate systems ofthe input device and the volumetric display. The range of the coordinatesystems is also obtained so that out-of-space conditions can bedetermined. Next, the system samples 332 positional outputs of the inputdevice, such as the digitizer, mouse, flock-of-birds, etc., to obtainthe location of the users input. This information can also includeinformation about where the user is pointing. This position (andorientation if desired) is mapped 334 into a 3D position within thevolumetric display using the coordinate system (and the orientationvector, if needed). The cursor or other position indicatingrepresentation, such as a virtual hand, is drawn 336 at the mappedposition with the volumetric display. The mapping may involvedetermining a position on the surface that is being touched by adigitizing stylus, projecting a ray into the enclosure from the touchposition where the ray is oriented by the pointing vector of the inputstylus and positioning the cursor at a variable or fixed position alongthe ray. Another mapping causes relative motion of a 3D input devicesuch as a glove to be imparted to a cursor when a motion function isactivated. Other mappings as discussed in the related applications arepossible.

[0038] The operations described with respect to FIG. 19, when adigitizing enclosure surface is the input configuration, allow the userto interact with a surface of a three-dimensional (3D) volumetricdisplay and affect the 3D content of the display responsive to theinteraction. The interaction involves the user manipulating the stylusin a sensing region of the digitizing grid, the mapping of the stylusposition to a 3D display position and the creation of a cursor at a 3Ddisplay position. The cursor, in one of a number of differentpossibilities, is created at a distance offset from a tip of the stylusalong a pointing vector of the stylus. The cursor can be used to performtypical functions such as selecting, painting, dragging/dropping, etc.

[0039] The present invention has been described with respect to inputconfigurations where commands are input through position sensing typedevices, such as a mouse, a pointer, touch sensitive surface, etc. It isalso possible to use other types of input configurations, such asnon-spatial configurations. One non-spatial input space or configurationis a conventional voice or speech recognition system. In thisconfiguration a voice command, such as “down” is recognized and theselected object or volume is moved accordingly. In this case down. Theobject is moved down in the display space at a constant slow rate untilit reaches the bottom or until another command, such as “stop” is inputand recognized. For user centric commands, such as “move closer”, a userposition sensing system inputs the user position, the position is usedto determine the relative position of the active object with respect tothe user or the vector pointing from user to the object. This vector isused to determine a direction for object movement. To move closer theobject is moved along the vector toward the user by moving in a negativedirection. Again the motion would continue until a blocking object isencountered or another command is recognized.

[0040] Another non-spatial input configuration uses non-speech sounds,such as tones from a conventional multifrequency tone generator. Eachmultifrequency combination corresponds to a command and a conventionaltone recognition system is used to convert the sounds to commands.

[0041] The input space or configuration could also use conventionaleye-tracking-head-tracking technologies alone or in combination withother input configurations.

[0042] The many features and advantages of the invention are apparentfrom the detailed specification and, thus, it is intended by theappended claims to cover all such features and advantages of theinvention that fall within the true spirit and scope of the invention.Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the invention tothe exact construction and operation illustrated and described, andaccordingly all suitable modifications and equivalents may be resortedto, falling within the scope of the invention.

What is claimed is:
 1. A system, comprising: a three-dimensional (3D)volumetric display output configuration having a display content; and aninput configuration coupled to the volumetric display outputconfiguration and allowing a user to affect the display content.
 2. Asystem as recited in claim 1, wherein the output configuration comprisesone of a dome, a cylinder, a cubical box and an arbitrary shape.
 3. Asystem as recited in claim 1, wherein the input configuration comprisesone of a 3D volumetric input space mapped to the 3D volumetric display,a planer 2D input space mapped to the 3D volumetric display, a planar 2Dinput space mapped to a planar 2D space within the 3D volumetricdisplay, and a non-planar 2D input space mapped to the 3D volumetricdisplay.
 4. A system as recited in claim 3, wherein the user producesinputs comprising one or directly with a hand, with a surface touchingdevice and with an intermediary device.
 5. A system as recited in claim3, wherein the input configuration further comprises one of an inputvolume adjacent to the display, an input volume surrounding the display,a digitizing surface covering a surface of the display, a digitizingsurface offset from the surface of the display, and an intermediarydevice used with the display.
 6. A system as recited in claim 5, whereinthe intermediary device comprises one of a stylus, a surface fittingmouse, a park able mouse, a multi-dimensional mouse, a movable inputdevice positioned on a bottom periphery of the display and a set ofidentical input devices positioned spaced around a bottom periphery ofthe display.
 7. A system as recited in claim 1, wherein the inputconfiguration comprises a non-planar 2D input space mapped to the 3Dvolumetric display.
 8. A system as recited in claim 7, wherein thenon-planar 2D input space comprises a digitizing surface covering asurface of the display and a digitizing stylus interacting with thesurface.
 9. A system as recited in claim 8, wherein the stylus has atip, and the stylus and digitizing surface produce a pointing vector anda cursor is created in the display at a fixed distance from the tipalong the vector.
 10. A system as recited in claim 1, wherein the inputconfiguration comprises a tracking system tracking a user.
 11. A systemas recited in claim 1, wherein the input configuration is non-spatial.12. A system as recited in claim 11, wherein the input configurationcomprises a voice recognition system allowing the use to affect thedisplay content using voice commands.
 13. A system as recited in claim1, wherein the input configuration and output configuration define aspatial correspondence between an input space and an output space.
 14. Asystem as recited in claim 13, wherein the spatial correspondencescomprises one of 3D to 3D, 2D planar to 3D, 2D planar to 2D planar andnon-planar 2D to 3D.
 15. A system as recited in claim 12, where theinput configuration, output configuration and the user define adynamically updatable spatial correspondence.
 16. A system, comprising:a dome shaped three-dimensional (3D) volumetric display having anenclosure surface; an input configuration comprising: a digitizingsurface covering the enclosure surface; a digitizing stylus having a tipand interacting with the digitizing surface under the control of a user;and a digitizing system coupled to the digitizing surface and outputtingnon-planar position coordinates and a pointing vector responsive to theinteraction between the stylus and the digitizing surface; and acomputer coupled between the display and the digitizing system,producing 3D content displayed in the display, mapping the non-planarposition coordinates to a 3D coordinate position in the display byoffsetting along the vector by an offset distance from the tip andaffecting the content at the 3D coordinate position.
 17. A method,comprising: interacting, by a user, with a surface of athree-dimensional (3D) volumetric display; and affecting the 3D contentof the display responsive to the interaction.
 18. A method as recited inclaim 12, wherein the display comprises a digitizing grid formed on thesurface, the interacting comprises the user manipulating the stylus in asensing region of the grid and the affecting comprises mapping a stylusposition in the sensing region to a 3D display position and creating acursor at a the 3D display position.